Environment application model

ABSTRACT

Various implementations disclosed herein include devices, systems, and methods that enable presenting environments including visual representations of multiple applications. In one implementation, a method includes presenting a view of an environment at an electronic device on a display. The view includes visual representations of a plurality of applications. The method further includes determining to provide a first application with access to a control parameter. The control parameter is configured to modify at least a portion of the view of the environment with virtual content, and the portion of the view includes at least a portion of content outside of a view of a visual representation associated with the first application. The method further includes restricting access to the control parameter by other applications which prevents the other applications from modifying the at least the portion of the view of the environment via the control parameter.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.17/732,837, filed on Apr. 29, 2022, which is a continuation of U.S.patent application Ser. No. 17/177,458 filed on Feb. 17, 2021, and whichclaims the benefit of U.S. Provisional Application Ser. No. 62/984,853filed Mar. 4, 2020, each of which is incorporated herein by reference inits entirety.

TECHNICAL FIELD

The present disclosure generally relates to the display of extendedreality (XR) environments, and in particular, to techniques forpresenting views of XR environments comprising visual representations ofmultiple applications.

BACKGROUND

Various devices are used to provide users with XR environments. Thecontent of such XR environments may be experienced by a user using amobile device, head-mounted device (“HMD”), or other device thatpresents the visual or audio features of the content. The experience canbe, but need not be, immersive, e.g., providing most or all of thevisual or audio content experienced by the user. The content can includepass-through video (e.g., in which real-world content is captured by acamera and displayed on a display with additional content) oroptical-see-through (e.g., in which real-world content is vieweddirectly or through glass and supplemented with displayed additionalcontent).

In some instances, the content can also include visual representationsof multiple applications. Some devices and software programs facilitatethe display of such content that includes visual representations ofmultiple applications. However, these devices and software programs donot provide sufficient features for resolving any conflicts arisingamong the multiple applications concerning views of XR environments.

SUMMARY

Various implementations disclosed herein relate to techniques forcustomizing an ambience of a XR environment presented to a user. In oneimplementation, a method includes presenting content representing a viewof an XR environment at an electronic device on a display of theelectronic device. The XR environment comprising visual representationscorresponding to a plurality of applications. A first application amongthe plurality of applications is designated as an elevated application.The elevated application is provided with access to a control parameterconfigured to modify an ambience of the XR environment. Otherapplications of the plurality of applications are precluded fromaccessing the control parameter while the first application isdesignated as the elevated application.

In accordance with some implementations, a non-transitory computerreadable storage medium has stored therein instructions that arecomputer-executable to perform or cause performance of any of themethods described herein. In accordance with some implementations, adevice includes one or more processors, a non-transitory memory, and oneor more programs; the one or more programs are stored in thenon-transitory memory and configured to be executed by the one or moreprocessors and the one or more programs include instructions forperforming or causing performance of any of the methods describedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIG. 1 illustrates a functional block diagram of an event sensor, inaccordance with some implementations.

FIG. 2 illustrates example two-dimensional (“2-D”) tiles of pixel eventsthat an event sensor may output for further processing, in accordancewith some implementations.

FIG. 3 is a block diagram of an example head-mounted device (HMD) inaccordance with some implementations.

FIG. 4 illustrates an example of image data that an image pipelinederives from pixel events output by an event sensor, in accordance withsome implementations.

FIG. 5 illustrates an example of a stateless approach for processingpixel events output by an event sensor.

FIG. 6 illustrates a comparison between example functional blockdiagrams of stateless and stateful processes.

FIG. 7 illustrates an example of a stateful approach for processingpixel events output by an event sensor.

FIG. 8 illustrates an example of a stateful approach of gaze trackingusing pixel events output by an event sensor.

FIG. 9 is a flow-chart illustrating an example of a method forpresenting views of XR environments comprising visual representations ofmultiple applications.

FIG. 10 is a block diagram of an example electronic device, inaccordance with some implementations.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsor variants do not include all of the specific details described herein.Moreover, well-known systems, methods, components, devices and circuitshave not been described in exhaustive detail so as not to obscure morepertinent aspects of the example implementations described herein.

Referring to FIG. 1 , an example operating environment 100 forimplementing aspects of the present disclosure is illustrated anddesignated generally 100. In general, operating environment 100illustrates an electronic device 120 involved in the display oftwo-dimensional (2-D) content or three-dimensional (3-D) content,including but not limited to, an extended reality (XR) environment to auser. an extended reality (XR) environment refers to a wholly orpartially simulated environment that people sense and/or interact withvia an electronic device. For example, the XR environment may includeaugmented reality (AR) content, mixed reality (MR) content, virtualreality (VR) content, and/or the like. With an XR system, a subset of aperson's physical motions, or representations thereof, are tracked, and,in response, one or more characteristics of one or more virtual objectssimulated in the XR environment are adjusted in a manner that comportswith at least one law of physics. As one example, the XR system maydetect rotational head movement and, in response, adjust graphicalcontent and an acoustic field presented to the person in a mannersimilar to how such views and sounds would change in a physicalenvironment. As another example, the XR system may detect rotational ortranslational movement of the electronic device presenting the XRenvironment (e.g., a mobile phone, a tablet, a laptop, or the like) and,in response, adjust graphical content and an acoustic field presented tothe person in a manner similar to how such views and sounds would changein a physical environment. In some situations (e.g., for accessibilityreasons), adjustments to characteristic(s) of graphical content in an XRenvironment may be made in response to representations of physicalmotions (e.g., vocal commands).

There are many different types of electronic systems that enable aperson to sense and/or interact with various XR environments. Examplesinclude head mountable systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmountable system may have one or more speaker(s) and an integratedopaque display. Alternatively, a head mountable system may be configuredto accept an external opaque display (e.g., a smartphone). The headmountable system may incorporate one or more imaging sensors to captureimages or video of the physical environment, and/or one or moremicrophones to capture audio of the physical environment. Rather than anopaque display, a head mountable system may have a transparent ortranslucent display. The transparent or translucent display may have amedium through which light representative of images is directed to aperson's eyes. The display may utilize digital light projection, OLEDs,LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, orany combination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In some implementations, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

In some implementations, the electronic device 120 is a handheldelectronic device (e.g., a smartphone or a tablet) configured to presentcontent representing a view (e.g., view 200 of FIG. 2 ) of an XRenvironment to a user. In some implementations, the electronic device120 is a head-mounted device (HMD) that a user wears. Such an HMD mayenclose the field-of-view of the user. In some implementations, an HMDis worn is a way that one or more screens are positioned to displaycontent representing a view of an XR environment in the field-of-view ofthe user.

In some implementations, electronic device 120 is configured to interactwith a controller (not shown) separate from electronic device 120 topresent content representing a view of an XR environment to a user. Insome implementations, the controller includes a suitable combination ofsoftware, firmware, or hardware to facilitate presentation of contentrepresenting a view of an XR environment to a user. In someimplementations, the controller is a computing device communicativelycoupled to electronic device that is local or remote relative to aphysical environment (e.g., scene 110) proximate to electronic device120. In one example, the controller is a local server located within aphysical environment proximate to electronic device 120. In anotherexample, the controller is a remote server (e.g., a cloud server,central server, etc.) located outside of the physical environmentproximate to electronic device 120. In some implementations, thecontroller is communicatively coupled with the electronic device 120 viaone or more wired or wireless communication channels (e.g., BLUETOOTH,IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

FIG. 2 is an example of presenting content 200 representing an XRenvironment on a display (e.g., the one or more displays 1006 of FIG. 10) of electronic device 120. The XR environment represented by content200 comprises depictions of physical elements within scene 110, such asa floor 112, a first wall 114, a second wall 116, and a table 130.Content 200 further comprises depictions of virtual elementscorresponding to multiple applications, such as a representation 210 ofa checkers application, a representation 220 of a media application, anda representation 230 of a weather application. Each representationdepicted in FIG. 2 comprises computer-generated content associated withor provided by a different application among the multiple applications.For example, representation 210 comprises computer-generated contentdepicting a virtual checkers board associated with the checkersapplication and representation 230 comprises computer-generated contentindicative of weather conditions provided by the weather application.

In one implementation, the XR environment represented by content 200provides a “virtual desktop” comprising visual representationscorresponding to multiple, separately executing applications. In thisimplementation, the virtual elements comprising those visualrepresentations are positioned or configured in the XR environment suchthat they are perceived to coexist with physical elements (e.g., table130) within the XR environment. For example, representation 210 andassociated computer-generated content 212 depicting virtual checkers arepositioned in the XR environment represented by content 200 such thatthose virtual elements are perceived as existing on a horizontal surface132 of table 130.

Content 200 generally corresponds to a view of an XR environment that auser of electronic device 120 may perceive before electronic device 120modifies an ambience of the XR environment, as described in greaterdetail below. “Ambience” generally refers to a character and/oratmosphere of an XR environment that a user perceiving the XRenvironment associates with the XR environment. Various factorscontribute to such ambience. For example, reducing a brightness levelassociated with an XR environment may contribute to a user associatingthat environment with a relaxing atmosphere whereas increasing thebrightness level may contribute to the user associating the XRenvironment with an energetic atmosphere. As another example,spatializing ambient sound for light rain all around a user in an XRenvironment may contribute to the user associating that environment witha relaxing atmosphere whereas spatializing ambient sound for smashingglass and collapsing structures may contribute to the user associatingthe XR environment with a threatening atmosphere.

In various implementations, an ambience unit (e.g., ambience unit 1042of FIG. 10 ) facilitates customization of an ambience of an XRenvironment presented to a user. To that end, the ambience unit exposesone or more control parameters for modifying the ambience of the XRenvironment to multiple applications associated with or providingcontent corresponding to the XR environment. An application interfaceunit (e.g., application interface unit 1044) minimizes a likelihood thatsuch control parameters receive conflicting inputs or commands byarbitrating control access to the one or more control parameters amongthe multiple applications.

One aspect of arbitrating control access involves the applicationinterface unit designating a particular application among the multipleapplications as an elevated application. In one implementation,designating a particular application as an elevated applicationcomprises the application interface unit identifying the particularapplication based on one or more identification criterion. Suchidentification criterion may include: a default configuration, apriority setting, a user preference, a user input, a characteristic ofthe particular application, a current context of the XR environment, acurrent context of a physical environment proximate to electronic device120, or a combination thereof.

Upon designating the particular application as the elevated application,the application interface unit provides the particular application withaccess to a control parameter configured to modify an ambience of the XRenvironment. While the particular application is designated as theelevated application, the application interface unit restricts orprecludes other applications among the multiple applications fromaccessing the control parameter. For example, the application interfaceunit may provide the particular application with read-write access tothe control parameter while providing the other applications with noaccess or read-only access to the control parameter. In oneimplementation, an elevated application is provided with exclusiveaccess to the control parameter. In one implementation, any conflictsarising among the multiple applications related to the control parameterare resolved based on which particular application among the multipleapplications is designated as the elevated application.

By providing an elevated application with access to a control parameter,the application interface unit enables the elevated application tointeract with the ambience unit to change a value of the controlparameter based on user input received via the elevated application. ANXR content creation unit (e.g., XR content creation unit 1046 of FIG. 10) detects changes to the value of the control parameter and modifies theambience of the XR environment accordingly. As illustrated by comparingFIG. 2 with any of FIGS. 3-4 , the control parameter to which theelevated application receives access may modify the ambience of the XRenvironment in various ways.

In one implementation, designating the particular application as theelevated application alters routing decisions of a user input detectionprocess concerning user input among the multiple applications. Forexample, the application interface unit may designate a firstapplication among the multiple applications as the elevated application.In this example, the user input detection process may route user input(e.g., gaze tracking data, selection gesture data, and the like) to asecond application of the multiple applications prior to the firstapplication being designated as the elevated application. However,subsequent to the first application being designated as the elevatedapplication, the user input detection process may route such user inputto the first application. In one implementation, the applicationinterface unit comprises the user input detection process.

A comparison between FIGS. 2 and 3 illustrates an example in which theelevated application receives access to a control parameter thatmodifies the ambience of the XR environment by adjusting a lightingcondition of the XR environment. In this example, the applicationinterface unit may designate the media application associated withrepresentation 220 as the elevated application while presenting content200 of FIG. 2 . Upon designating the media application as the elevatedapplication, the application interface unit provides the mediaapplication with access to the control parameter that adjusts thelighting condition of the XR environment.

In FIG. 2 , a value of the control parameter may define a first state inwhich the lighting condition of the XR environment comprises a maximumbrightness level. User input received via the media application maychange that value of the control parameter to a value that defines asecond state in which the lighting condition of the XR environmentcomprises a minimum brightness level. Responsive to detecting thatchange to the value of the control parameter, the XR creation unitgenerates content 300 in which the lighting condition of the XRenvironment is adjusted to the second state.

A comparison between FIGS. 2 and 4 illustrates an example in which theelevated application receives access to a control parameter thatmodifies the ambience of the XR environment by adjusting an immersionlevel of the XR environment. In this example, the media applicationassociated with representation 220 is again designated by theapplication interface unit as the elevated application while presentingcontent 200 of FIG. 2 . Upon designating the media application as theelevated application, the application interface unit provides the mediaapplication with access to the control parameter that adjusts theimmersion level of the XR environment.

In FIG. 2 , a value of the control parameter may define a firstinversion level in which a user may equally perceive both physicalelements and virtual elements in the XR environment. User input receivedvia the media application may change that value of the control parameterto a value that defines a second inversion level in which the user'sability to perceive virtual elements is reduced. Responsive to detectingthat change to the value of the control parameter, the XR creation unitgenerates content 400 in which the immersion level of the XR environmentis adjusted to the second immersion level. In content 400, the user'sability to perceive a representation 430 of the weather application isreduced relative to its representation 230 in content 200 while theuser's ability to perceive depictions of physical elements (e.g., table130) remains static. Other virtual elements of content 400, such as arepresentation 410 of the checkers application, associatedcomputer-generated content 412, and a representation 420 of the mediaapplication are also less perceivable than their counterparts in content200.

In the examples discussed above with respect to FIGS. 3 and 4 , theapplication interface unit provides an elevated application with accessto a control parameter that modifies aspects of the XR environment'sambience that generally pertain to the XR environment at large. Theambience of the XR environment that such control parameters modify maybe referenced as “environmental ambience”. In some implementations, theambience unit exposes one or more user interface parameters that modifyaspects of the XR environment's ambience that are more localized to aparticular application among the multiple applications. The ambience ofthe XR environment that such user interface parameters modify may bereferenced as “local ambience”. Each user interface parameter isconfigured to modify a local ambience pertaining to portions of the XRenvironment occupied by a visual representation of the particularapplication or portions of the XR environment within a predefinedproximity to that representation.

By way of example, the application interface unit may provide theweather application, as an elevated application, with access to acontrol parameter corresponding to an environmental audio ambience. Inthis example, the weather application may modify the control parameterto spatialize weather-related ambient sound (e.g., rainfall) all arounda user in an XR environment. In an implementation, one or morenon-elevated applications (e.g., the media application) may eachspatialize audio from a respective direction of a correspondingrepresentation concurrent with a modification of a control parametercorresponding to an environmental audio ambience.

As another example, the application interface unit may provide theweather application, as an active application, with access to a userinterface parameter corresponding to a local audio ambience. In thisexample, the weather application may modify the user interface parameterto spatialize weather-related ambient sound (e.g., rainfall) from adirection of its corresponding representation in an XR environment. Inan implementation, one or more non-active applications (e.g., the mediaapplication) may each spatialize audio from a respective direction of acorresponding representation concurrent with a modification of a userinterface parameter corresponding to an local audio ambience.

In implementations with user interface parameters configured to modifylocal ambiences, another aspect of arbitrating control access involvesthe application interface unit designating a particular applicationamong the multiple applications as an active application. In oneimplementation, the active application is identified by the applicationinterface unit based on a detected location of a user position withinthe XR environment. In one implementation, the active application isidentified by the application interface unit responsive to receivinguser input that designates the particular application as the activeapplication. In one implementation, the active application is identifiedby the application interface unit responsive to detecting an ongoinginteraction between the particular application and a user.

Upon designating the particular application as the active application,the application interface unit provides the particular application withaccess to a user interface parameter configured to modify a localambience of the XR environment. While the particular application isdesignated as the active application, the application interface unitprecludes other applications among the multiple applications fromaccessing the user interface parameter. For example, the applicationinterface unit may provide the particular application with read-writeaccess to the user interface parameter while providing the otherapplications with no access or read-only access to the user interfaceparameter. In one implementation, an active application is provided withexclusive access to the user interface parameter.

In one implementation, designating the particular application as theactive application alters routing decisions of a user input detectionprocess concerning user input among the multiple applications. Forexample, the application interface unit may designate a firstapplication among the multiple applications as the active application.In this example, the user input detection process may route user input(e.g., gaze tracking data, selection gesture data, and the like) to asecond application of the multiple applications prior to the firstapplication being designated as the active application. However,subsequent to the first application being designated as the activeapplication, the user input detection process may route such user inputto the first application. In one implementation, the applicationinterface unit comprises the user input detection process.

A comparison between FIGS. 2 and 5 illustrates an example in which anactive application receives access to a user interface parameter thatmodifies a local ambience of the XR environment. In this example, theweather application associated with representation 230 is designated bythe application interface unit as the active application whilepresenting content 200 of FIG. 2 . Upon designating the weatherapplication as the active application, the application interface unitprovides the weather application with access to the user interfaceparameter that modifies the local ambience of the XR environment.

In FIG. 2 , a value of the user interface parameter may define a firststate in which the lighting condition of representation 230 comprises amaximum brightness level. User input received via the weatherapplication may change that value of the user interface parameter to avalue that defines a second state in which the lighting condition ofrepresentation 230 comprises a minimum brightness level. Responsive todetecting that change to the value of the user interface parameter, theXR creation unit generates content 500 in which the lighting conditionof that representation of the weather application is adjusted to thesecond state, as illustrated by representation 530. Of note, the localambience of the XR environment in the example of FIG. 5 is modifiedwithout modifying the environmental ambience of the XR environment.

A comparison between FIGS. 5 and 6 illustrates an example in which anactive application receives access to a user interface parameter that isconfigured to modify a local ambience of the XR environment based on afunctionality provided by the active application. In this example, theapplication interface unit provides the weather application with accessto the user interface parameter that modifies the local ambience of theXR environment based on the functionality provided by the activeapplication. While presenting content 500 of FIG. 5 , the weatherapplication receives weather data indicative of a weather forecastchange. In accordance with a functionality provided by the weatherapplication, its representation is updated to reflect the weatherforecast change, as illustrated by representation 630.

In FIG. 5 , a value of the user interface parameter may define a firststate in which a portion of the XR environment within a predefineddistance of representation 530 is populated with image data depictingscene 110. In an implementation, the image data depicting scene 110 isobtained using an image sensor (e.g., an image sensor within the imagesensory system 1012 of FIG. 10 ) of electronic device 120. In FIG. 6 ,user input received via the weather application may change that value ofthe user interface parameter to a value that defines a second state inwhich the portion of the XR environment within that predefined distanceis populated with computer-generated content corresponding to thefunctionality provided by the weather application. Responsive todetecting that change to the value of the control parameter, the XRcreation unit generates computer-generated content 632.

A comparison between FIGS. 6 and 7 illustrates an example in which aparticular application among multiple applications may concurrently bedesignated as an active application and an elevated application. In thisexample, the weather application associated with representation 630 isagain designated by the application interface unit as the activeapplication while presenting content 600 of FIG. 6 . Moreover, thisexample presumes that the media application associated withrepresentation 220 remains designated as the elevated application, asdiscussed above with reference to FIG. 2 . While presenting content 600,the application interface unit may designate the weather application asboth the active application and the elevated application. For example,the application interface unit may reevaluate the one or moreidentification criterion discussed above with reference to FIG. 2 anddesignate the weather application as the elevated application responsiveto that reevaluation.

Upon designating the weather application as the elevated application,the application interface unit provides the weather application withaccess to the control parameter that adjusts the lighting condition ofthe XR environment. In one implementation, the application interfaceunit removes access to the control parameter by the media applicationwhile the weather application is designated as the elevated application.In this example, the application interface unit provides the weatherapplication with access to a control parameter configured to modify anappearance of one or more portions of the XR environment upondesignating the weather application as the elevated application. In oneimplementation, other applications (e.g., the media application and thechess application) are precluded or restricted from modifying the one ormore portions of the XR environment while the weather application isdesignated as the elevated application.

In FIG. 6 , a value of the control parameter configured to modify theappearance of the one or more portions of the XR environment may definea first state in which that portion is populated with image datadepicting scene 110. In an implementation, the image data depictingscene 110 is obtained using an image sensor (e.g., an image sensorwithin the image sensory system 1012 of FIG. 10 ) of electronic device120. In FIG. 7 , user input received via the weather application maychange that value of the control parameter to a value that defines asecond state in which that portion of the XR environment is populatedwith computer-generated content. In one implementation, thecomputer-generated content corresponds to a functionality provided bythe elevated application. Responsive to detecting that change to thevalue of the control parameter, the XR creation unit generatescomputer-generated content 712, 714, and 716.

In one implementation, user input received via a particular applicationamong multiple applications concurrently designated as an activeapplication and an elevated application may modify both a local ambienceassociated with the particular application and an environmentalambience. For example, the user input discussed above with respect toFIG. 7 that updates the value of the control parameter may also updatethe value of the user interface parameter discussed above with respectto FIG. 6 . In this instance, responsive to detecting that change to therespective values of the control parameter and the user interfaceparameter, the XR creation unit may also generate computer-generatedcontent 632 and overlay representation 630 in content 700 withcomputer-generated content 632. As a result, that user input may modifyboth a local ambience associated with the weather application and anenvironmental ambience of the XR environment represented by content 700.

A comparison between FIGS. 2 and 8 illustrates an example in which aparticular application designated as an active application is providedwith supplementary access to another application. In this example, theweather application associated with representation 230 is designated bythe application interface unit as the active application whilepresenting content 200 of FIG. 2 . Upon designating the weatherapplication as the active application, the application interface unitprovides the weather application with supplementary access to anotherapplication. For example, the application interface unit may direct userinput received via the weather application to an application programminginterface (API) of a reference application, such as a searchapplication.

In FIG. 2 , user input indicative of a search query concerning anextended weather forecast may be received via the weather applicationwhile presenting content 200. Responsive to detecting the user input,the application interface unit may direct the user input to the API ofthe reference application. Upon receiving the user input at the API, thereference application may transition from an unlaunched state to alaunched state and direct the search query to a search engine associatedwith the reference application. When the reference application receivesa corresponding search response, the XR creation unit generates content800, in which a representation 810 of the reference application ispopulated with data obtained in the corresponding search response.

FIG. 9 is a flowchart illustrating an exemplary method 900 of providinga view of an XR environment, according to some implementations. In someimplementations, the method 900 is performed by a device (e.g.,electronic device 120 of FIG. 1 ). The method 900 may be performed at amobile device, HMD, desktop, laptop, server device, or by multipledevices in communication with one another. In some implementations, themethod 900 is performed by processing logic, including hardware,firmware, software, or a combination thereof. In some implementations,the method 900 is performed by a processor executing code stored in anon-transitory computer-readable medium (e.g., a memory).

At block 902, the method 900 includes presenting, at an electronicdevice, content representing a view of an XR environment on a display ofthe electronic device. The XR environment comprising visualrepresentations corresponding to a plurality of applications. At block904, the method 900 includes designating a first application among theplurality of applications as an elevated application. In oneimplementation, designating the first application as the elevatedapplication comprises identifying the first application based on adefault configuration, a priority setting, a user preference, a userinput, a characteristic of the first application, a current context ofthe XR environment, a current context of a physical environmentproximate to the electronic device, or a combination thereof. In oneimplementation, designating the first application as the elevatedapplication alters routing decisions of a user input detection processamong the plurality of applications.

At block 906, the method 900 includes providing the elevated applicationwith access to a control parameter configured to modify an ambience ofthe XR environment. In one implementation, the control parameter isconfigured to modify the ambience by adjusting a lighting condition ofthe XR environment from a first state to a second state. In oneimplementation, the control parameter is configured to modify anappearance of a portion of the XR environment. In one implementation,other applications among the plurality of applications are precludedfrom modifying the appearance of the portion of the XR environment whilethe first application is designated as the elevated application. In oneimplementation, the control parameter is configured to modify theambience by adjusting an immersion level of the XR environment from afirst level to a second level.

At block 908, the method 900 includes precluding other applications ofthe plurality of applications from accessing the control parameter whilethe first application is designated as the elevated application. In oneimplementation, the elevated application has exclusive access to thecontrol parameter. In one implementation, any conflicts arising amongthe plurality of applications related to the control parameter areresolved based on which of the plurality of applications is designatedas the elevated application.

In one implementation, method 900 further includes modifying theambience of the XR environment based on user input received via theelevated application. In one implementation, method 900 further includesdesignating a second application among the plurality of applications asthe elevated application. In one implementation, method 900 furtherincludes providing the second application with access to the controlparameter responsive to designating the second application as theelevated application. In one implementation, method 900 further includesremoving access to the control parameter by the first application whilethe second application is designated as the elevated application.

In one implementation, method 900 further includes designating a thirdapplication among the plurality of applications as an activeapplication. In one implementation, method 900 further includesproviding the active application with access to a user interfaceparameter configured to modify a functionality provided by the activeapplication. In one implementation, method 900 further includesprecluding other applications of the plurality of applications fromaccessing the user interface parameter while the third application isdesignated as the active application. In one implementation, designatingthe third application as the active application alters routing decisionsof a user input detection process among the plurality of applications.In one implementation, the active application is identified based on adetected location of a user position within the XR environment.

In one implementation, the ambience is an environmental ambiance andmethod 900 further comprises modifying a local ambience of a visualrepresentation of the third application based on user input received viathe third application without modifying the environmental ambience. Inone implementation, method 900 further includes designating the thirdapplication as the elevated application and the method further comprisesmodifying both the local ambience of the visual representation of thethird application and the environmental ambience based on user inputreceived via the third application.

FIG. 10 is a block diagram of an example electronic device 120 inaccordance with some implementations. While certain specific featuresare illustrated, those skilled in the art will appreciate from thesubject matter disclosed herein that various other features have notbeen illustrated for the sake of brevity, and so as not to obscure morepertinent aspects of the implementations disclosed herein.

To that end, as a non-limiting example, in some implementationselectronic device 120 includes one or more processors 1002 (e.g.,microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, or thelike), one or more communication interfaces 1004 (e.g., USB, FIREWIRE,THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA,GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or the like type interface), oneor more displays 1006, one or more I/O devices and sensors 1008, one ormore programming (e.g., I/O) interfaces 1010, one or more image sensorsystems 1012, a memory 1020, and one or more communication buses 1014for interconnecting these and various other components.

In some implementations, the one or more communication interfaces 1004can include any device or group of devices suitable for establishing awired or wireless data or telephone connection to one or more networks.Non-limiting examples the one or more communication interfaces 1004include a network interface, such as an Ethernet network adapter, amodem, or the like. A device coupled to the one or more communicationinterfaces 1004 can transmit messages to one or more networks aselectronic or optical signals.

In some implementations, the one or more displays 1006 are configured topresent a user interface. In some implementations, the one or moredisplays 1006 correspond to holographic, digital light processing(“DLP”), liquid-crystal display (“LCD”), liquid-crystal on silicon(“LCoS”), organic light-emitting field-effect transitory (“OLET”),organic light-emitting diode (“OLED”), surface-conductionelectron-emitter display (“SED”), field-emission display (“FED”),quantum-dot light-emitting diode (“QD-LED”), micro-electromechanicalsystem (“MEMS”), or the like display types. In some implementations, theone or more displays 1006 correspond to diffractive, reflective,polarized, holographic, etc. waveguide displays. In one example, theelectronic device 120 includes a single display. In another example, theelectronic device 120 includes a display for each eye. In someimplementations, the one or more displays 1006 are capable of presentingXR content.

In some implementations, the one or more I/O devices and sensors 1008are configured to provide a human to machine interface exchangingcommands, requests, information, data, and the like, between electronicdevice 120 and a user. To that end, the one or more I/O devices 1008 caninclude, but are not limited to, a keyboard, a pointing device, amicrophone, a joystick, and the like. In some implementations, the oneor more I/O devices and sensors 1008 are configured to detect or measurea physical property of an environment proximate to electronic device120. To that end, the one or more I/O devices 1008 can include, but arenot limited to, an IMU, an accelerometer, a magnetometer, a gyroscope, athermometer, one or more physiological sensors (e.g., blood pressuremonitor, heart rate monitor, blood oxygen sensor, blood glucose sensor,etc.), one or more microphones, one or more speakers, a haptics engine,one or more depth sensors (e.g., a structured light, a time-of-flight,or the like), and/or the like.

In some implementations, the one or more programming (e.g., I/O)interfaces 1010 are configured to communicatively couple the one or moreI/O devices 1008 with other components of electronic device 120. Assuch, the one or more programming interfaces 1010 are capable ofaccepting commands or input from a user via the one or more I/O devices1008 and transmitting the entered input to the one or more processors1002.

In some implementations, the one or more image sensor systems 1012 areconfigured to obtain image data that corresponds to at least a portionof a scene local to electronic device 120. The one or more image sensorsystems 1012 can include one or more RGB cameras (e.g., with acomplimentary metal-oxide-semiconductor (“CMOS”) image sensor or acharge-coupled device (“CCD”) image sensor), monochrome camera, IRcamera, event-based camera, or the like. In various implementations, theone or more image sensor systems 1012 further include illuminationsources that emit light, such as a flash.

The memory 1020 can include any suitable computer-readable medium. Acomputer readable storage medium should not be construed as transitorysignals per se (e.g., radio waves or other propagating electromagneticwaves, electromagnetic waves propagating through a transmission mediasuch as a waveguide, or electrical signals transmitted through a wire).For example, the memory 1020 may include high-speed random-accessmemory, such as DRAM, SRAM, DDR RAM, or other random-access solid-statememory devices. In some implementations, the memory 1020 includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid-state storage devices. The memory 1020 optionallyincludes one or more storage devices remotely located from the one ormore processing units 1002. The memory 1020 comprises a non-transitorycomputer readable storage medium. Instructions stored in the memory 1020may be executed by the one or more processors 1002 to perform a varietyof methods and operations, including the technique for presenting a viewof an XR environment comprising visual representations of multipleapplications described in greater detail above.

In some implementations, the memory 1020 or the non-transitory computerreadable storage medium of the memory 1020 stores the followingprograms, modules and data structures, or a subset thereof including anoptional operating system 1030, an XR experience module 1040, and one ormore applications 1050. In one implementation, the one or moreapplications 1050 may include any combination of the applicationsdiscussed above with respect to FIGS. 2-8 . For example, the one or moreapplications may include: the checkers application, the mediaapplication, the weather application, and/or the reference application.The operating system 1030 includes procedures for handling various basicsystem services and for performing hardware dependent tasks.

In some implementations, the XR experience module 1040 is configured tocreate, edit, present, or experience XR environments. To that end, invarious implementations, the XR experience module 1040 includes anambience unit 1042, an application interface unit 1044, and an XRcontent creation unit 1046.

In some implementations, the ambience unit 1042 facilitatescustomization of an ambience of an XR environment presented to a user.In one implementation, the ambience unit 1042 facilitates suchcustomization by exposing one or more control parameters for modifyingan ambience of the XR environment to multiple applications associatedwith or providing content corresponding to the XR environment. In oneimplementation, the ambience unit 1042 facilitates such customization byexposing one or more user interface parameters configured to modify alocal ambience of the XR environment. To that end, in variousimplementations, the ambience unit 1042 includes instructions or logictherefor, and heuristics and metadata therefor.

In some implementations, the application interface unit 1044 isconfigured to designate a particular application among multipleapplications as an elevated application and provide the elevatedapplication with access to a control parameter. In one implementation,the application interface unit 1044 is configured to designate aparticular application among the multiple applications as an activeapplication and provide the active application with access to a userinterface parameter. To that end, in various implementations, theapplication interface unit 1044 includes instructions and/or logictherefor, and heuristics and metadata therefor.

In some implementations, the XR content creation unit 1046 is configuredto generate content representing a view of an XR environment. In oneimplementation, the XR content creation unit 1046 is configured togenerate content that modifies an environmental ambience of the XRenvironment responsive to detecting a change in a value of a controlparameter. In one implementation, the XR content creation unit 1046 isconfigured to generate content that modifies a local ambience of the XRenvironment responsive to detecting a change in a value of a userinterface parameter. To that end, in various implementations, the XRcontent creation unit 1046 includes instructions or logic therefor, andheuristics and metadata therefor.

Although the ambience unit 1042, the application interface unit 1044,and the XR content creation unit 1046 are shown as residing on a singledevice (e.g., electronic device 120), it should be understood that inother implementations, any combination of the ambience unit 1042, theapplication interface unit 1044, and the XR content creation unit 1046may be located in separate computing devices.

FIG. 10 is intended more as functional description of the variousfeatures which are present in a particular implementation as opposed toa structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 10 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one implementation to another and, in some implementations, dependsin part on the particular combination of hardware, software, or firmwarechosen for a particular implementation.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor value beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

It will also be understood that, although the terms “first,” “second,”etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularimplementations only and is not intended to be limiting of the claims.As used in the description of the implementations and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “or” as used hereinrefers to and encompasses any and all possible combinations of one ormore of the associated listed items. It will be further understood thatthe terms “comprises” or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, or components, but do not preclude the presence or addition ofone or more other features, integers, steps, operations, elements,components, or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

The foregoing description and summary of the invention are to beunderstood as being in every respect illustrative and exemplary, but notrestrictive, and the scope of the invention disclosed herein is not tobe determined only from the detailed description of illustrativeimplementations but according to the full breadth permitted by patentlaws. It is to be understood that the implementations shown anddescribed herein are only illustrative of the principles of the presentinvention and that various modification may be implemented by thoseskilled in the art without departing from the scope and spirit of theinvention.

1. A method comprising: at an electronic device with a display:presenting a view of an environment on the display, the view comprisinga first visual representation of content provided by a first applicationand a second visual representation of content provided by a secondapplication; providing the first application with access to enable thefirst application to modify the view of the environment outside of thefirst visual representation of the content provided by the firstapplication; and restricting the second application by preventing thesecond application from modifying the view of the environment outside ofthe second visual representation of the content provided by the secondapplication.
 2. The method of claim 1, further comprising: modifying theview of the environment based on user input received via the firstapplication.
 3. The method of claim 1, wherein providing the firstapplication with access to enable the first application to modify theview of the environment outside of the first visual representation ofthe content provided by the first application comprises identifying thefirst application based on a default configuration, a priority setting,a user preference, a user input, a characteristic of the firstapplication, a current context of the environment, a current context ofa physical environment proximate to the electronic device, or acombination thereof.
 4. The method of claim 1, wherein modifying theview of the environment outside of the first visual representation ofthe content comprises modifying an ambience of the environment.
 5. Themethod of claim 4, wherein modifying the ambience of the environmentcomprises adjusting a lighting condition of the environment from a firststate to a second state.
 6. The method of claim 1, wherein otherapplications are precluded from modifying an appearance of the view ofthe environment while the first application is enabled to modify theview of the environment outside of the first visual representation ofthe content provided by the first application.
 7. The method of claim 1,wherein modifying the view of the environment comprises adjusting animmersion level of the environment from a first level to a second level.8. The method of claim 1, further comprising: determining to provide thesecond application with access to enable the second application tomodify the view of the environment outside of the second visualrepresentation of the content provided by the second application;removing access to the first application to modify the view of theenvironment outside of the first visual representation of the contentprovided by the first application; and providing the second applicationwith access to enable the second application to modify the view of theenvironment outside of the second visual representation of the contentprovided by the second application responsive to determining to providethe second application with access.
 9. The method of claim 1, whereinproviding the first application with access to enable the firstapplication to modify the view of the environment outside of the firstvisual representation of the content provided by the first applicationcomprises altering routing decisions of a user input detection processamong a plurality of applications.
 10. The method of claim 1, furthercomprising: designating a third application as an active application;providing the active application with access to a user interfaceparameter configured to modify a functionality provided by the activeapplication; and precluding remaining applications from accessing theuser interface parameter while the third application is designated asthe active application.
 11. The method of claim 10, wherein the activeapplication is identified based on a detected location of a userposition within the environment.
 12. The method of claim 10, furthercomprising: modifying a third visual representation of the contentprovided by the third application based on user input received via thethird application without modifying the first visual representationassociated with the first application.
 13. The method of claim 12,further comprising: providing the third application with access toenable the third application to modify the view of the environmentoutside of the third visual representation of the content provided bythe third application; and modifying an ambience associated with thethird visual representation of the content provided by the thirdapplication and modifying an ambience associated with the first visualrepresentation associated with the first application based on user inputreceived via the third application.
 14. The method of claim 1, whereinmodifying the view of the environment outside of the first visualrepresentation of the content provided by the first applicationcomprises modifying an environmental audio ambience by spatializingambient sound around a user position in the environment.
 15. The methodof claim 1, wherein modifying the view of the environment outside of thefirst visual representation of the content provided by the firstapplication comprises modifying an environmental audio ambience byspatializing audio from a respective direction of a correspondingapplication with access to modify the view of the environment.
 16. Asystem comprising: an electronic device with a display; a processor; anda computer-readable storage medium comprising instructions that uponexecution by the processor cause the processor to perform operations,the operations comprising: presenting a view of an environment on thedisplay, the view comprising a first visual representation of contentprovided by a first application and a second visual representation ofcontent provided by a second application; providing the firstapplication with access to enable the first application to modify theview of the environment outside of the first visual representation ofthe content provided by the first application; and restricting thesecond application by preventing the second application from modifyingthe view of the environment outside of the second visual representationof the content provided by the second application.
 17. The system ofclaim 16, wherein the instructions that upon execution by the processorfurther cause the processor to perform the operations: modifying theview of the environment based on user input received via the firstapplication.
 18. The system of claim 16, wherein providing the firstapplication with access to enable the first application to modify theview of the environment outside of the first visual representation ofthe content provided by the first application comprises identifying thefirst application based on a default configuration, a priority setting,a user preference, a user input, a characteristic of the firstapplication, a current context of the environment, a current context ofa physical environment proximate to the electronic device, or acombination thereof.
 19. The system of claim 16, wherein theinstructions that upon execution by the processor further cause theprocessor to perform the operations: modifying an ambience of theenvironment by adjusting a lighting condition of the environment from afirst state to a second state.
 20. A non-transitory computer-readablestorage medium, storing program instructions executable by a processorto perform operations comprising: presenting a view of an environment onthe display, the view comprising a first visual representation ofcontent provided by a first application and a second visualrepresentation of content provided by a second application; providingthe first application with access to enable the first application tomodify the view of the environment outside of the first visualrepresentation of the content provided by the first application; andrestricting the second application by preventing the second applicationfrom modifying the view of the environment outside of the second visualrepresentation of the content provided by the second application.